Modeling and Interpreting Multimodal Inputs: A Semantic Integration Approach

Modern user interfaces can take advantage of multiple input modalities such as speech, gestures, handwriting... to increase robustness and flexibility. The construction of such multimodal interfaces would be greatly facilitated by a unified framework that provides methods to characterize and interpret multimodal inputs. In this paper we describe a semantic model and a multimodal grammar structure for a broad class of multimodal applications. We also present a set of grammarbased Java tools that facilitate the construction of multimodal input processing modules, including a connectionist network for multimodal semantic integration. This research was sponsored by the DARPA under Department of the Navy, Naval Research Office under grant number N00014-93-1-0806, and Project GENOA under grant number 97047-ISX/ SOW-600066. Views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the DARPA or the U.S. Government. Modeling and Interpreting Multimodal Inputs: A Semantic Integration Approach 2